Dither is Better than Dropout for Regularising Deep Neural Networks
نویسنده
چکیده
Regularisation of deep neural networks (DNN) during training is critical to performance. By far the most popular method is known as dropout. Here, cast through the prism of signal processing theory, we compare and contrast the regularisation effects of dropout with those of dither. We illustrate some serious inherent limitations of dropout and demonstrate that dither provides a far more effective regulariser which does not suffer from the same limitations.
منابع مشابه
Parallel Dither and Dropout for Regularising Deep Neural Networks
Effective regularisation during training can mean the difference between success and failure for deep neural networks. Recently, dither has been suggested as alternative to dropout for regularisation during batch-averaged stochastic gradient descent (SGD). In this article, we show that these methods fail without batch averaging and we introduce a new, parallel regularisation method that may be ...
متن کاملObject Classification using Deep Convolutional Neural Networks
The objective of this research project is to explore the impact on performance by varying architectures of deep neural networks. Deep neural networks have resurged in interest by researchers when, in 2012, Krizhevsky et al. submitted a deep convolutional neural network to the ILSVRC (ImageNet Large Scale Visual Recognition Challenge) and achieved significantly-higher results than the entire com...
متن کاملAdaptive dropout for training deep neural networks
Recently, it was shown that deep neural networks can perform very well if the activities of hidden units are regularized during learning, e.g, by randomly dropping out 50% of their activities. We describe a method called ‘standout’ in which a binary belief network is overlaid on a neural network and is used to regularize of its hidden units by selectively setting activities to zero. This ‘adapt...
متن کاملDropout training for Hidden Unit CRFs
A very commonly faced issue while training prediction models using machine learning is overfitting. Dropout is a recently developed technique designed to counter this issue in deep neural networks and has also been extended to other algorithms like SVMs. In this project, we formulate and study the application of Dropout to Hidden Unit Conditional Random Fields (HUCRFs). HUCRFs use binary stocha...
متن کاملUnderstanding Representations and Reducing their Redundancy in Deep Networks
Neural networks in their modern deep learning incarnation have achieved state of the art performance on a wide variety of tasks and domains. A core intuition behind these methods is that they learn layers of features which interpolate between two domains in a series of related parts. The first part of this thesis introduces the building blocks of neural networks for computer vision. It starts w...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
- CoRR
دوره abs/1508.04826 شماره
صفحات -
تاریخ انتشار 2015